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The top quark is being studied in unprecedented detail 
with the large data samples from Run II of the Fermilab 
Tevatron collider. Since the top quark is by far the 
most massive known fermion, with a coupling to the 
Higgs field of order unity, these studies may shed light 
on the mechanism of electroweak symmetry breaking and 
provide hints of new physics. Within the standard model 
(SM), the top quark coupling to the bottom quark and 
the W boson (tWb) has the V — A form of a left-handed 
vector interaction. We consider a more general form 
for the tWb coupling to allow for departures from the 
SM [l[ . We look for physics beyond the SM in the form 
of right-handed vector couplings or left- or right-handed 
tensor couplings, described by the effective Lagrangian 
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including operators up to dimension five [2| 



-1^(/ T ^ + f^Pn)tW~ + h.c. ,(1) 



where Myy is the mass of the W boson, q is its 
four-momentum, Vtb is the Cabibbo-Kobayashi-Maskawa 
matrix element 0, and P L = (l-7 5 )/2 (Pr = (l+7 5 )/2) 
is the left-handed (right-handed) projection operator. In 
the SM, the left-handed vector coupling form factor is 
fy = 1, the right-handed vector coupling form factor 
is fy ~ 0; an< i the tensor coupling form factors are 
fr = /t = 0- We assume real coupling form factors, 
implying CP conservation, and a spin-^ top quark which 
decays predominantly to Wb. 

An alternative parameterization of anomalous 
couplings through effective operators has been proposed 
recently 0, [j| . The anomalous coupling limits presented 
in this letter can be translated into the operator 
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where A is the scale of the new physics and v = 246 GeV 
is the scale of electroweak symmetry breaking. Cr 3+S , 
CS, Cj|^ and C^y are constants for dimension-six 
gauge-invariant effective operators for third generation 
quarks, involving the Higgs field (<fi), the W boson, up- 
type (u) and down-type (d) quarks. The constants C are 
assumed to be real. 

Indirect constraints on the magnitude of the right- 
handed vector coupling and tensor couplings exist from 
measurements of the b — > sj branching fraction Q. 
General unitarity considerations require the anomalous 
tensor couplings to be less than 0.5 |7|. While the b — > sj 
limits are tighter than the direct limits presented in this 
Letter, they include assumptions that are not required 
here, in particular that there is no new physics affecting 
the b quark other than anomalous tWb couplings. Direct 
constraints on anomalous tWb couplings have been 
obtained from previous DO analyses [a |9j and from an 
analysis of LHC results (loj . 

This Letter describes a combination of recent W boson 
helicity 11 1 and single top quark Q measurements, 



using the same procedure as in a previous combination 
of W boson helicity with single top quark information 
in DO data Q. Deviations from the SM expectation 
in the coupling form factors manifest themselves in 
two distinct ways that are observable at DO: (i) by 
altering the fractions of W bosons from top quark decays 
produced in each of the three possible helicity states, 
and (ii) by changing the rate and kinematic distri- 
butions of electroweak single top quark production. We 
translate W boson helicity fractions [lH into form factors 
using the general framework given in Ref. [12J. By 



combining these with the single top quark anomalous 
couplings analysis Q, we obtain posterior probability 
density distributions for the anomalous coupling form 
factors. Three separate scenarios are investigated using 
the same dataset, for fy , fjt, and f^ ■ In each scenario we 
investigate the anomalous coupling form factor and the 
SM coupling form factor fy simultaneously and set the 
other two anomalous coupling form factors to zero. We 
form a two-dimensional posterior density as a function of 
two coupling form factors and then marginalize over the 
SM coupling to obtain a 95% C.L. limit on the anomalous 
coupling. 



This analysis is based on data collected with the 
DO detector corresponding to an integrated 

luminosity of 5.4 fb _1 . For the W boson helicity 
analysis, tt events are selected in both the lepton plus 
jets (ti — > W + W~bb — > Ivqq'bb, requiring a lepton, 
missing transverse energy and at least four jets) and 



dilepton (tt -> W + W~bb 



'v'bb, requiring two 



leptons, missing transverse energy and at least two jets) 
channels (llj . 

We use the alpgen leading-order Monte Carlo (MC) 
event generator [TtJ , interfaced to PYTHIA [3| , to model 
tt events as well as VF+jets and Z+jcts background 
events. We generate it events with both SM V — A 
and V + A couplings, and reweight these to model any 
given W boson helicity state. We use the CTEQ6L1 
parton distribution functions flii ] and set the top quark 
mass to 172.5 GeV, consistent with the world average 
top mass [2C|. The response of the DO detector is 
simulated using GEANT [2l| . The presence of additional 
pp interactions is modeled by overlaying the simulation 
with data events, selected from random beam crossings 
matching the instantaneous luminosity profile in the 
data. The background from multijet production, where 
a jet is misidentified as an isolated electron or muon, is 
modeled with data events containing lepton candidates 
that pass all of the lepton identification requirements 
except one, but otherwise resemble the signal events. We 
use MC to model the smaller background from dibosons. 
The SM single top quark background is modeled using 
the COMPHEP MC event generator [22| normalized to 
theory predictions [13] . In the W boson helicity analysis, 
the possible presence of anomalous couplings does not 
significantly modify the small background from single top 
quark production. A multivariate likelihood discriminant 
that uses both kinematic and b quark lifetime information 
distinguishes tt events from background, separately in the 
lepton plus jets and dilepton channels. A requirement 
on the likelihood selects 1431 lepton plus jet events 
and 319 dilepton events with expected backgrounds of 
404 ± 32 and 69 ± 10 events, respectively, where the 
uncertainty includes both statistical and background 
modeling components. 

We determine the fractions of W bosons with left- 
handed, longitudinal, and right-handed helicity (/_, /o, 
and f + . respectively). The SM predicts /_ = 30%, 
fo = 70%, and /+ w ©(lO" 4 ) 0. The fractions are 
measured in a fit to the distribution of the angle 8*, 
where 8* is the angle between the direction opposite 
to the top quark and the direction of the down-type 
fermion (charged lepton or down-type quark) from the 
decay of the W boson, both in the W boson rest frame. 
A binned maximum likelihood fit compares the cos 8* 
distribution of the selected events to expectations from 
each W boson helicity state and the background. In the 
lepton plus jets channel, each possible assignment of the 
four leading jets in the event is considered to reconstruct 
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the two top quarks in the event, based on the x 2 of a 
kinematic fit and the compatibility between the assigned 
jet flavor and b quark lifetime information. For the 
W boson that decays hadronically, we do not attempt to 
determine which of the daughter jets corresponds to the 
up-type quark. Rather we select one jet at random. Since 
this introduces a sign ambiguity, we can only distinguish 
the longitudinal helicity from the other two states and 
can no longer distinguish left-handed and right-handed 
helicity states. In the dilepton channel, we determine the 
momenta of the two neutrinos using an algebraic solution. 
Since the system is kinematically underconstrained, we 
assume a value for the top quark mass of 172.5 GeV to 
perform the kinematic reconstruction. We vary both the 
longitudinal and right-handed helicity fractions fo and 
/+ in the fit and find the relative likelihood of any set 
of helicity fractions being consistent with the data. The 
result is presented in Fig.[IJ which also demonstrates how 
non-SM values for the coupling form factors could alter 
the W boson helicity fractions. 



>1.2 
1 

0.8 
0.6 
0.4 
0.2 




-0.2 



DO, 5.4 fb" 1 


• Best-fit value 

★ SM value 

— 68%C.L 

— 95%C.L. 
■ f v *0 

- fV*0 


_ 










'\ 


+ , , , 




I i i i I i i i I i i i 



-0.2 



0.2 0.4 0.6 0.8 



1.2 



FIG. 1: (color online) Likelihood contours at the 68% C.L. 
and the 95% C.L. as a function of W boson helicity fractions. 
Statistical uncertainties and systematic uncertainties that 
are uncorrelated with the single top quark measurement are 
included. The squares, triangles and upside-down triangles 
show fy, and /|? varying in fifty equal-size steps such that 
their ratio to fy goes from zero to ten-to-one. The dashed 
triangle denotes the physically allowed region. 

The result is interpreted in terms of the coupling form 
factors in Fig. [2j which shows that the W boson helicity 
measurement only constrains ratios of the coupling form 
factors and not their magnitude. These distributions 
provide one of the inputs to the combined constraint on 
the coupling form factors. 

The other input to the form factor constraint comes 
from the search for anomalous tWb couplings in the single 



top quark final state. Both i-channel (the exchange of a 
W boson between a light quark and a heavy quark) and s- 
channel (the production and decay of a virtual W boson) 
modes contribute to single top quark production at the 
Tevatron. Single top quark production was observed 
by the CDF and DO collaborations [H, [2(|, and the 
i-channel mode was also isolated by the DO collabo- 
ration 



27] 



Both the single top quark production cross section 
and kinematic distributions are modified by anomalous 
couplings. The single top quark cross section may also 
differ from the SM prediction because \Vtb\ < 1, but that 
is not considered here. We assume that single top quark 
production proceeds exclusively through the tWb vertex 
and not through the exchange of a new particle. Wc 
also assume that \V t d\ 2 + \V ts \ 2 <C |Vt&| 2 , i.e., top quark 
production and decay through light quarks is negligible. 

The single top quark anomalous couplings analysis 
selects events in which the top quark decays to a IF boson 
and a b quark, followed by the decay of the W boson to 
an electron or muon, and a neutrino. The final state 
contains two or three jets, one from the top quark decay, 
one produced together with the top quark, and possibly a 
third jet from initial-state or final-state gluon radiation. 
The event selection is identical to that in the anomalous 
coupling single top quark analysis || and the SM single 
top quark analysis [28|, except that events with four jets 
are removed from the sample to avoid overlap with the 
W boson helicity analysis. One or two of the jets are 
required to be ^-tagged, i.e., identified as originating from 
B hadrons j^. To increase the search sensitivity, the 
data are divided into four independent analysis channels 
based on jet multiplicity (2 or 3), and number of 6-tagged 
jets (1 or 2). 

Wc use Bayesian neural networks (BNN) [30[ to 
discriminate between the single top quark anomalous 
coupling signal and the backgrounds. For each of the 
three coupling scenarios, the signal in the BNN training 
consists of only that particular anomalous single top 
quark couplings sample while the background in the 
training consists of all SM backgrounds plus SM single 
top quark events. The main background contributions to 
the single top quark analysis are those from IF+jets, ti 
and multijet production. The background modeling and 
normalization procedures are the same as in the W boson 
helicity analysis. The ti contribution to the background 
is small and is modeled by simulated SM ti events and 
normalized to the theoretical cross section j3l|. The 
effect of anomalous couplings on the ti background is 
negligible. We model the single top quark signal using 
the COMPHEP MC event generator [22| where anomalous 
tWb couplings are considered in both the production and 
decay of the top quark. 

We use the four- vectors of the reconstructed final state 
particles in the BNN training (transverse momentum px, 
pscudorapidity 77, angle Acf> with respect to the lepton, 
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and the mass of each jet), i.e.. twelve variables for events 
with two jets and sixteen variables for events with three 
jets. We add four angular variables that are particularly 
sensitive to the anomalous couplings. These are cosines 
of angles between various final state objects in the top 
quark rest frame. 

The BNN output is used in a Bayesian analysis that 
determines a posterior density as a function of the 
anomalous coupling and the SM coupling, separately for 
each scenario. Figure [3] shows the probability density 
distributions from the single top quark anomalous 
couplings search, and the middle column of Table Q] gives 
the anomalous coupling form factor limits obtained from 
the single top quark anomalous couplings analysis alone. 
These differ slightly from those given in Ref. [U due to 
the exclusion of the 4-jet sample. 

We account for all systematic uncertainties and 
their correlations among different analysis channels, 
and sources of signal or background, in the two 
analyses. Systematic uncertainties in the W boson 
helicity measurement arc detailed in Ref. (ill ]. They 
arise from finite MC statistics and uncertainties on the 
top quark mass, jet energy scale, and MC models of 
signal and background. Variations in these parameters 
can change the measurement in two ways: by altering 
the estimate of the background (i.e., if the background 
selection efficiency changes) and by modifying the shape 
of the cos 6* templates. Systematic uncertainties on the 
tt normalization do not affect the measurement. We also 
assign a systematic uncertainty to account for differences 
between the input /o and /+ values and the average fit 
values in pseudo-experiments. 

Systematic uncertainties on the signal and background 
models in the single top quark anomalous couplings 
analysis are estimated using the methods described in 
Ref. J28|. The dominant sources of uncertainty are the 
jet energy scale, 6-tag modeling, and MC models of 
signal and background, with smaller contributions from 
background normalizations, top quark mass, and object 
identification. 

Uncertainties that only affect the W boson helicity 
measurement are MC statistics and the tt cos 9* template 
modeling uncertainty. Uncertainties that only affect 
the single top quark anomalous coupling analysis are 
those related to signal modeling and background normal- 
ization, including luminosity, object reconstruction, and 
6-tag modeling. 

We use a Bayesian statistical analysis [3^ to combine 
the W boson helicity result with that of the single top 
quark anomalous couplings analysis. The likelihood from 
the W boson helicity analysis shown in Fig. [2] is used as 
a prior to the analysis of single top anomalous couplings 
analysis. For each anomalous coupling form factor 
scenario (fy, /jf, and fj$), we compare the corresponding 
BNN output for data with the sum of backgrounds and 
two signal models, the anomalous coupling model and the 



SM (fy). In the f£ scenario the two amplitudes interfere 
for single top quark production, which is taken into 
account through a superposition of three signal samples: 
one with only left-handed vector couplings, one with 
only left-handed tensor couplings, and one with both 
coupling form factors set to one (which also includes 
the interference term). For tt production all interference 
terms are accounted for properly in all three scenarios. 

We then compute a likelihood as a product over 
all separate analysis channels. We assume Poisson 
distributions for the observed counts and use Gaussian 
distributions to model the uncertainties on the signal 
acceptance and background yields, including correlations 
of systematic uncertainties. The uncertainties are 
evaluated through MC integration in an ensemble of 
200,000 samples. Each sample has the same data distri- 
bution but signal and background contributions that are 
shifted by the systematic uncertainties, i.e., the signal 
and background shapes and normalizations as well as 
the prior from the W boson helicity change for each 
sample. The final posterior is the ensemble average of 
all individual posteriors. 

The two-dimensional posterior probability density is 
computed as a function of \fy\ 2 and \fx\ 2 , where fx is 
fy, f£, or /j? . These probability density distributions 
including both W boson helicity and single top quark 
anomalous coupling information are shown in Fig. |4] We 
observe no significant anomalous contributions. 

We compute 95% C.L. upper limits on the anomalous 
form factors by integrating over the left-handed 
vector contribution to obtain one-dimensional posterior 
probability densities. The limits are given in Table HI 

TABLE I: Observed upper limits on anomalous tWb couplings 
at 95% C.L. from W boson helicity assuming fy — 1, from 
the single top quark analysis, and from their combination, for 
which no assumption on fy is made. 



Scenario 


only 


only 


combination 




W helicity 


single top 






0.62 


0.89 


0.30 


\rt\ 2 


0.14 


0.07 


0.05 


\m 2 


0.18 


0.18 


0.12 



Table U also shows the limits obtained from only the 
W boson helicity analysis with the additional assumption 
that fy = 1. Compared with the results obtained using 
only the single-top search, the combination improves 
the limits on the form factors significantly because the 
individual analyses provide complementary information. 

The 95% C.L. limits on the coupling operators in the 
operator notation based on Eq. [2] are |C^' 3+3 ^| < 14.7, 
|Cf|| < 18.0, \C 3 d 3y\ < 2.5, and \C 3 Jy\ < 4.1, assuming 
a new physics scale of A = 1 TcV. The limit on cj?' 3+3 ^ 
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is obtained from the fy scenario filter by setting fy = 
and integrating the resulting \fy\ 2 posterior density 



starting at \fy\ 2 



1 to find the 95% C.L. limit on the 



anomalous contribution. Limits for the other operators 
are obtained from the corresponding form factor limits. 
These limits are a significant improvement over previous 
limits. A separate analysis of Tevatron and early LHC 
results provides limits on anomalous couplings that 
appear stronger than those presented here even though 
it uses less information. This is mainly due to the use 
of priors that are flat in the coupling rather than the 
coupling squared as is done here. 

In summary, we have presented a study of tWb 
couplings that combines W boson helicity measurements 
in top quark decay with anomalous couplings searches 
in the single top quark final state, thus using all 
currently applicable top quark measurements by DO. 
We find consistency with the SM and set 95% C.L. 
limits on anomalous tWb couplings. Our limits represent 
significant improvements over previous DO results beyond 
the increase in luminosity. 
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FIG. 2: (color online) Likelihood density as a function of tWb coupling form factors, for (a) right-vector vs. left-vector couplings, 
(b) left-tensor vs. left-vector couplings, and (c) right-tensor vs. left-vector couplings, using information from the W boson 
helicity measurement only. All systematic uncertainties are included. Each color corresponds to a contour of equal likelihood 
density. 
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FIG. 3: (color online) Form factor posterior density distribution for (a) right-vector vs. left-vector couplings, (b) left-tensor vs. 
left-vector couplings and (c) right-tensor vs. left-vector couplings, using information from the single top quark analysis only, 
for events with two or three jets. All systematic uncertainties are included. 
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FIG. 4: Posterior density distribution for the combination of W boson helicity and single top quark measurements for (a) 
right-vector vs. left-vector form factors, (b) left-tensor vs. left-vector form factors and (c) right-tensor vs. left-vector form 
factors. All systematic uncertainties are included. 



